What is claimed is: 



CLAIMS 



1 . A method comprising: 

maintaining a current state of auxiliary information regarding a sequence of video 
frames, the sequence of video frames being encoded as a video bit stream having video 
frame data for each respective video frame of the sequence of video frames; 

comparing the current state of auxiliary information with auxiliary information 
regarding a current video frame of the sequence of video frames to determine differential 
information; and 

annotating the differential information to the video bit stream as an annotation to 
the video frame data for the current video frame. 

2. The method of Claim 1, wherein the current state of auxiliary information 
indicates the latest encoded information in the bit stream and is maintained in a state 
storage memory. 

3. The method of Claim 1, wherein the auxiliary information comprises 
additional non-visual information regarding video frames. 

4. The method of Claim 1, wherein the auxiliary information comprises 
video processing information regarding video frames. 

5. The method of Claim 1, further comprising encoding the differential 
information before annotating the differential information. 
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6. The method of Claim 5, wherein the encoded differential information 
comprises a list of parameters, the parameters being described in a tag lookup table. 

7. The method of Claim 1, wherein annotating the differential information 
comprises extending the video bit stream format to include the differential information. 

8. The method of Claim 1, wherein annotating comprises annotating only if 
the differential information indicates a change from the current state of the auxiliary 
information. 

9. The method of Claim 1 , further comprising gathering the video 
information regarding the current video frame. 

10. The method of Claim 9, wherein the differential information comprises the 
difference between the current state of the auxiliary information and the gathered 
information. 

11. A machine-readable medium having stored thereon data representing 
instructions which, when executed by a machine, cause the machine to perform 
operations comprising: 

maintaining a current state of auxiliary information regarding a sequence of video 
frames, the sequence of video frames being encoded as a video bit stream having video 
frame data for each respective video frame of the sequence of video frames; 

comparing the current state of auxiliary information with auxiliary information 
regarding a current video frame of the sequence of video frames to determine differential 
information; 
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annotating the differential information to the video bit stream as an annotation to 
the video frame data for the current video frame. 

12. The medium of Claim 1 1 , wherein the current state of auxiliary 
information indicates the latest encoded information in the bit stream and is maintained 
in a state storage memory. 

13. The medium of Claim 1 1, wherein the auxiliary information comprises 
additional non-visual information regarding video frames. 

14. The medium of Claim 11, further comprising encoding the differential 
information as a list of parameters, the parameters being described in a tag lookup table, 
before annotating the differential information. 

15. The medium of Claim 14, wherein annotating the differential information 
comprises extending the video bit stream format to include the differential information. 

16. An apparatus comprising: 

a collector to receive auxiliary information regarding a sequence of video frames 
and to maintain a current state of the auxiliary information, the sequence of video frames 
being encoded as a video bit stream having video frame data for each respective video 
frame of the sequence of video frames; 

a comparator to receive the auxiliary information and compare the current state of 
the auxiliary information with auxiliary information regarding a current video frame of 
the sequence of video frames to determine differential information; and 
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an annotator to store the differential information in the video bit stream by 
extending the original bit stream format. 

17. The apparatus of Claim 16, wherein the auxiliary information comprises 
additional non-visual information regarding video frames. 

18. The apparatus of Claim 16, wherein the auxiliary information comprises 
video processing information regarding video frames. 

19. The apparatus of Claim 16, wherein the collector receives the sequence of 
video frames. 

20. The apparatus of Claim 16, further comprising a state storage memory 
coupled to the collector to store the current state of the auxiliary information. 

2 1 . The apparatus of Claim 1 6, further comprising an encoder to encode the 
differential information as auxiliary information for the current frame data; 

22. The apparatus of Claim 16, wherein the encoded differential information 
comprises a list of parameters, the parameters being described in a tag lookup table. 

23. An apparatus comprising: 

a collector to receive auxiliary information regarding a sequence of video frames 
and to maintain a current state of the auxiliary information, the sequence of video frames 
being encoded as a video bit stream having video frame data for each respective video 
frame of the sequence of video frames; 

a state storage memory coupled to the collector to store the current state of the 
auxiliary information; 
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a comparator to receive the auxiliary information and compare the stored current 
state of the auxiliary information with auxiliary information regarding a current video 
frame of the sequence of video frames to determine differential information; and 

a tag lookup table defining settable parameters; 

an annotator to store the differential information using the settable parameters in 
the video bit stream by extending the original bit stream format. 

24. The apparatus of Claim 16, wherein the auxiliary information comprises 
additional non-visual information regarding camera geometry and identification for the 
video frames. 

25. The apparatus of Claim 16, wherein the auxiliary information comprises 
video processing information regarding descriptions and camera positions for the video 
frames. 
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